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concerning the evaluation of instructional products. Two basic { 
problems, which must be solved before adequate empirical data may be: 
accumulated, concerning D&E models are: (1) just what constitutes a ' 
development or evaluation model, and how can one identify and 
adequately describe such models; and (2) what are the actual purposes 
for using development and evaluation models in materials development. 
As researchers begin to address these questions and use them’to focus 
their work, batter evidence will come to light regarding the utility 
of using models in the development and evaluation of instructional 
products. (MB) ’ 
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THE STATUS OF RESEARCH ON MODELS 

" PRODUCT DEVELOPMENT AND EVALUATION 
Models are apparently being proffered and used rather freely -todgy in 
ai development and evaluation of instructional products. This paper pbttrs a 
eeitioal look at the current research basé supporting the use of theed mals 
and identifies two basic problems with which researchers will have ito cope as 
they begin to add to the existing research opiates, : 

In assessing the research on the utility of development ‘and evaluation 
models in the creation of instructional products, one finds him bie fn much 
the same position as Norman Hamm in his review of the efficacy and . 
sioihinnoatdas of research on the mental health of children amm, 1974). 

He notes that his tnethod of inquiry is "limited, if os somewhat paradoxical. 
What is offered is a rational analysis of our empirical eles as it applies 
to practical Savin [p. 9)." Echoed here is his call for more attention to a 
science of science, including research performed on veidowsth policy. 

As a first step, then, one nea to examine just /what is meant -" the 


use of evaluation and development models in product development. ° 


Purposes of Models 


Individuals writing on the nature and use of models in science (c.f. 


( ' { 
Kaplan, 1964, Black, 1973) have noted several uses made of the-term.. Black, 


for example, discusses how the term "model" has been used in science’ to 
denote (1) scale models, (2) analogue models in which thare is an isomorphic 
relationship between the model and the original entity, (3) exemplar models 

= \ 


which outline designs or conceptions of things worth of imitation, and 
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(4) theoretical models in which one uses an area better understood in order to. 


‘ solve a particular. problem. It will bé argued here that the term "models" 
- typically used in educational cine and evaluation appears to refer 

to the third definition above, i.e.., these development and ee models 
are simply designs of exemplary processes or procedures. They do not 
appear, for instance, to oe since they generally lack 8 
explanatory’ and/or predictive puitinn. They simply appear. to be axenilariy, 
of amiciité ot commendable operating procedures and are probably bést 
ylewed as alternative, descriptive conceptualizations of development cu 
évaluation processes. Black mentions such a use when he Attstifeete models — 
as a "convenience of exposition. ./that/ may also help us to: notice what other- 
: wise may be overlooked, to shift the relative emphasis attached to details--in 
short, to see. new connections {[p. 496]." This aa of the term "models" seems 
particularly appropriate to the models used to es evaluation vee where 

* there are several competing conceptualizations of siencasniaes evaluation pro- 
cedures, each based on differing assumptions se eines concerning the 
evaluation process itself. Development models, however, frequently appear to 


be as much managerial strategies as helpful conceptual paradigms of any 


particular development process. , The characterization of models as heuristic 


“N 


devices for use in specific contexts suggests a conceptual difficulty in generali- 
zing or being prescriptive about the use of particular mgdels across varying 


‘circumstances and situations. - . 


o : 


Increased ‘Attention to Models 


does not permit a full review of all development and evaluation models cur- | 


hihere has, been an increased proliferation of models in the last few 
4 


years. from the research and development communities and 4 pecyscanttan 


increased ‘interest in ‘them by the federal funding agencies. AioGagh space 


rently being proffered, a few words of summary seem necessary. ‘, 

“Probably the most current and comprehensive discussion.of models and 
procedures for instructional development can be found in 7 (1973). Both 
Baker and Schutz (1970) have referred to two ‘major development emphases: _ 
those nang to do with change support process and those having to do with Z 
a product) development process. They suggest that there may be differing 
development models and ip appropriate to the different emphasis of 
these twd, approaches. Baker nti some reason to doubt whether there 
could be any generalizable development models for change support processes ' 
in pointing out that development procedures are ‘highly idiosyncratic to partic- 


ular. contexts. Merrill and Boutwell (1973), in reviewing books and manuals 


‘ 


designed to give guidance to developers of instructional systems, have noted | 
that though| they vary in terminology, the procedures recommended all 


basically include the following components: behavioral objectives, pre-test, 


 ———— 


K 
instructional activities, post-test, and revision based on empirical results. 


There: appears to be less uniformity in the field of evaluation models 


s 


% ; ; 
‘with its proliferation of models such as the CIPP Model (Stufflebeam, 1968), 
the Countenance Model (Stake, 1967), the Discrepancy Model (Provis, 1971), 


the Pathways Model (Scriven, undated), ete. Worthen and Sanders (1973) 


: 


overview, 
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_ provide the most current comparison of the major evaluation models, though 


the collection of readings by Taylor and Cowley (1972) also provides a good 


al 


on 
In addition to the shovedada attention being paid by the research com- 
munity to the development and use of models, ‘there apparently has been an 
increaging tendency =e models a a ident of seman si operation- 
alizing nave development and evaluation policy. Drafts of policy statements 


* . | 
from the. National. Institute of Education (NIE) (c.f. NIE 1974a, NIE’ 1974b) 


and documents submitted to NIE for possible policy adoption, (c.f. Institute . 


for Educational Development, 1973) suggest that NIE is taking a more prescrip- 
tive stance toward the use of evaluation and development models in its 
monitoring of instructional development work. ! 


This increased prescriptiveness seems to emanate, at least in part, 


from NIE's response to a report to Congress made by the Comptroller 


‘General concerning a review of educational laboratory and research develop- 


ment center programs (Comptroller choise of the United States, 1973), In 
its response to this report, NIE,agreed, among other things, to (1) sagas 
contractors to state objectives in terms jof the ‘specific saandiiens changes 
that are expected, (2) to produce emptrfeal evidence of the accomplishments 
of students who use the materials, and (3) to define more’clearly the various 


stages of product development and testing (Comptroller General of the United 


. States, 1973). 


By NIE's own admission,this increased prescriptiveness is not based on 


empirical evidence, 
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Generally we have lacked both the, data base and the under- 
standing of system dynamics needed for effective rational | 
policy making. In their absence, policies have been determined 
‘on the basis of the resolution of cross-pressures, analogies 
drawn from other fields, and the imagination and foresight of 

a few individuals [NIE, 1973, p. 65]. 


Just how true this statement must be is evidenced by a brief review of the 


past research in the area of evaluation and develépment models. 


Past Research on Models | 

Whether one looks at the overall rationale and empirical evidence sup- 
porting the use of develigmient models or the justification of specific compo- 
nents of development procedures, the picture is equally bleak. Merrill and 


Boutwell (1973), after noting that almost all development procedures reviewed 


seemed to be based upon (a) the specification of objectives in terms of, observ- 


able student behaviors, (b) the use of criterion-referenced rather than norm- 


‘ io 
referenced measurement procedures, and (c) the verification of instructional 
products by empirical test results, note that "the paucity of research on these 
premises suggest that they are considered axiomatic. Numerous propositions 


have been suggested, but’only a handful of rather poorly executed studies have 


tested them [p. 95)." Baker (1973) also notes that much of the literature in 
, instructional development prescribes, on the basis of faith alone, "Jegions of 


procedures.'' Noting a lack of empirical research in this area, she summarizes 
by saying, 


\~ .. the field of educational development requires inquiry into the 
kinds of development procedures which are effective in particular 
contexts for various classes of organizations. From the current / 
state of:the art, one would suppose that minimally such procedures 
would require specification, field testing and revision as a 
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foundation for development work. The possible ways in which each 
_of these points might be translated into practice must be explored 
(p.. 277) oc 
” Though the specification of behavioral objectives may be thought to be 
basic to most development models, the Yresults of research on the effective- 


nese of using behavioral objectives as a means of improving:teaching, guiding 


evaluation, or in the facilitation of learning, seems to be inconsistent and 


: v ‘ 
. equivocal at best (Duchastal ard Merrill, 1973; McKeachie, 1974; Stake, 1973). 


There are a number of conceptual and technical challenges which the criterion 


referenced test siwiicaientt must meet before any development model could 
. safely ldepend upon their widespread development and use (Ebel, 1973; Klein | 
"and Kosecoff, 1973). Further, Baker 973) indicates that there is Sor 7 
_ indicative ideas that certain types of media contribute to a better learning 
environment than other types of media or no media at all. She suggests that 
developers themselves have to make evaluation studies of probably contanding 
media to determine their eolntive effectiveness for the varkiouler objectives 
of interest. And finally, it is becoming increasingly clear that the adoption 


of even the most basic "laws of learning" and hierarchial relations which 


has been a very stgnificant feature of dévdispinent models is largely without 
empirieal justification as they are now conceptualized (McKeachie, 1974a, 
McKeachie, 1974b, White, 1973). 

iinet at the more global components of development models reveals 
the same lack of research foundation. Baker (1973) notes that while there 
’ 


are many guides for writing performatice specifications, the research in 


support of such practice is not to be found. She indicates that the influence 
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of, specification on the purveyors of instruction has not been demonstrated in 
. 4 
controlled experimental studies. Although there are many guidelines on how 


to write performance specifications, there is no, research today to indicate 

the utility of such procedures to other developers. She notes that the question 
, 

of the generalizability of specific revision, procedures to othér tasks and other 


s 


subject matters have only "been touched upon bitefin, 


Popham (1969) notes that: this same situation prevails with research con- 


“cerning the development of curriculum materials, 


. ++. Studies for the revision process to- improve the quality of curriculum 

’ materials |ave not been frequéhtly conducted. There seems to be an 

’ overriding faith in the idea that materials revised to be consistent with : - 
empirical tryout data will become better. This has’ not been clearly 
demonstrated. Certainly the manner in which revisions can be made 
most efficiently has “not been carefully treated [p. 335]. 


’ 


The state of the art appears to be no. better for evaluation: models. 


Worthen (1972) has pointed to the spars#ty, of research on the phenomena of 


. evaluation itself. He notes that present evaluation models "... represent 


be ¢ 
(individually and collectively) any inadequate knowledge base for the guidance of 


educational evaluation (p. 3)." He notes there is little or no information 
about the relative alficacy of. alternative evaluation techniques and calls for 
research on the evaluation models themselves as well pa on the critical 
¢omponents of — sodels, such ag the’ obtrusiveness of data viglhaetton 
techniques ea the educational ‘phenomena being evaluated, and the differential 
effectiveness of feedback provided from evaluattgn under differing conditions 
and scheduling. He also notes the need for studies to igentifp the most 


effective means of identifying goals and conducting needs assessments. a i 
« 
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; Stake and Denny ‘(1969) have written about the lack of available techniques 


eo . 
‘and knowledge base for the conduct of evaluation.. They discuss, for example, 


‘ 


“the lack of techniques for presenting goals and priorities and the relative lack , 
of good tisteutbentation for assessing instructional materials in sladeeiom: 
tnatruction. For example, sehiaalteiny that there are many traditional 
observational schedulés and techniques, they point to how little information is 
available on how validly these techniques can be used and how widely. In 


addition, they note the lack of adequate instructional assgssment schedules 


_and eall for research and development which pays attention tb the "'.. . widely 
:, nee 
reported phenomenon of experimenter bias and the obtrusiveness of the class- 


‘.room observer [p. 38]. They also indicate a need for simple ways of 


meaningfully describing what the instructional program consists of and what 


Me * 7 a. 4s 
the students do in and after training. Finally, they discuss evaluation's 


_eurrent lack of ability to show how students perform and to measure student's 


performance in evaluation-related contexts. 


Thus, in evaluation as well as development, there is not only a lack of 


, 


research evidence on the utility of various models or conceptual approaches, 


but a lack of evidence on the effectiveness of component procedures and the 
r) 1 
basic asdumptions within the models themselves. It appears that, at this 
. * 
time, these models cannot be considered anything more than convenient con- : 


4 


ceptual: méchanisms or, at most, value-based (not empirically based) claims 


. about what constitutes "good" development and evaluation procedures. ° de 


There may be a need at the federal policy level for increased, prescrip- 


tiveness of the use of evaluation and development models, but such 


‘Al 


prescriptiveness is currently not based on the results of empirical ‘research. 


In such ‘cases, therefore, it ought to be made clear that these prescriptive 


recommendations are made from policymakers' past experience or value basis 


» 


or from untested hypotheses concerning the appropriateness of models selected 


from fields other than education. 
: : ‘ . 
Problems in Conducting Research 


The case has been made thus far that a great deal of research needs to’ 


be conducted upon evaluation and development procedures before empirically 


. 


based policy. can be developed. There are some basic conceptual problems, ; 
however, which must be solved before adequate research can be conducted 


upon the development and evaluation mddels themselves. 


° 
‘ 


One basic problem is ‘the conceptual delineation of” jet what constitutes 
a development or evaluation src and how one can identify and adequately 
‘describe such models. Merrill and Boutwell (1973) have discussed the lack 
ot the systematic ideriifiontion of the yariables being manipulated by instruc- 
tional developers and have proposed a taxonomy which would enable such . 
identification. They note. that i is often difficult for instructional material 
developers to describe their products unambiguously and thereby profit by the. 
“work of others in the field. In addition, they cite the difficulties jisereseitael 
researchers have in describing their experimental treatments adequately, > 
Merrill and Bostwell have proposed 4 task variable taxonomy as‘a first step 
in: resolving some of these djfficulties. Schutz (1970) notes that . 
The ‘ scope of the shenmeenor eon ae development,' has yet to 


be realized. While there is a vague ‘realization that it’ should be 
possible to translate available relevant knowledge into a form that 
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permits improved educational practice, it is erroneous to assume that~ 
the means for doing this are perfectly dlear [p. 39]. 


Schute further argues that the realm of educational Asivelonnend is still too 
-young ahd rapidly eoning to make {t even feasible to attempt to describe the 
state of the art. 
Whilé the development domain is highly sophisticated and well dif-— 
ferentiated in many areas, in education it has until very recently 


either been regarded ‘as mnecoeseyy or confounded with research 
[p. 40]. , 


That this problem of lack of conceptual clarity is not confined to development 
» 
is evidenced in the discussion by Worthen and Sanders (1973) of the continuing 
disagreement within the field about the most appropriate and useful definition 
of evaluation. (c.f. Scriven, 1972). o. 
yg 
A second major problem to be overcome before addressing such concerns 
as (1) the utility of various development and gyaluation models, (2) the compa- a 


tibility of given evaluation and development models in a particular context, 


and (3) the interaction of particular evaluation and development models with 2 


the nature of the product, is the lack of conceptual clarity about the actuab 
purposes for using development and evaluation mncdale in materials Acvvitoaininsit 
- An save to the question: what is an atinctive development or evaluation 
‘model? * requires that one first consider "effective for what purpose?" Medels 
have been used frequently in the past and apparently without a need on the 
part of the users we a strong empirical base. Development and evaluation. 
models may currently be being used because of a belief (hei leobvionsty no 
empirical pvidenctt thet they will result in the efficient production of high 
quality products and result in efficient and relevant assessment of the orth 
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all- 
“of these, products. } This view of i ol use of models, however, is baséd upon 


en egmoational technalogy a and it is possible . that models are used 


i 


not for. their utility in terms of educational technology, but because they 


serve as efficient political and managerial means of directing and controlling 


x 


development work, 


Michael Apple (1972) has suggested that the use of systems management 


procedures (in many ways similar to the eurrent use of development and 
evaluation models) actually prévides means of increasing certainty and control: . 
It should be made clear, then, that systems approaches are not 
essentially neutral, nor are they performing a "scientific" function. 
By tending to cause its users and the other publics involved to 
ignore certain possible fundamental‘problems with schools as insti- 
tutions, systems management also acts to generate and channel 


political sentiments supportive of the _exisitng modes of access to . 
knowledge and a il (p. 15]. ; 


Obviously iin may be chosen by researchers for their heuristic and 
conceptual features but used by administrators because of their monitoring 
and er capabilities. If one dancing that these models am being ; 
prescribed at the ‘federal policy level, however, more for their managerial 
capabilities than thelr scieatinael technology base, | then the previous literature 

sihuivlad fe incomplete. The criteria for good development. and evaluation 
models then lie not only in their utility in providing the most technically . 
sound products, but in their itility for monitoring and guiding Gui datelogeticn” 
process itself as well. *This leads — to view the nthe of models as another. 
‘ype of. management strategy ma suggents ‘research reviews and further 
saiwinih fn the areas ‘of * ee and educational management and not in the 


. area of instructional technology. 
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The appropriate criteria and even most germane fields of research for 

development and evaluation models are therefore inextricably tied to the pur-. 
pose or use ,that such models haye in the instructional development process. 
-Presumably. models can fulfill multiple purposes or roles in“ the akeubiennl 
development process; indeed, Scriven (1967) sometime ago discussed the 
multiple ta evaluation can generally play. If, howevor, development and 
evaluation models are ‘cunals alternative heuristic Conceptualizations of their 
respective processes, then’ the assessment of their validity or utility must be 
made in terms of the value perceptions of the relevant individuals involved, 
Nae one conducts value research. If, on the other hand, these models} are / 
to be ned as validated procedures for producing the technologically best 
instructional products, then considerably more research needs to be con- 
‘ducted in the field of instructional technology. Finally, if these models are 
to es tnd as ‘ managerial control and monitoring mechanism then research 
in the area. of industrial and educational management is most relevant. At 
present there appears to be little clarity in the field of instructional materials 
development regarding the actual purposes developmeht and evaluation models 

are being used for and more importantly, the actual purposes for which’ such 
models were originally designed. 


In summary, then, it appears that there is practically no solid empirical 


a ch . 
evidence to justify the prescriptive use of development and evaluation models 


in instructional product development -- at least from an instructional tech- 
‘ 


‘nology standpoint. . Furthermore, before such a research base can be provided, 


two significant conceptual problems have to be addressed by the researcher. 
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First, how does me define, identify, and describe development and evaluation 
models per se, and secondly, sai does one determine when re instructional 
Sikeston perspective is indeed more relevant than an fphaiklal sesinsecrdent | 
or value research perspective? Perhaps as. researchers begin to address 
these questions and use them to focus their work more clearly, better evidence 


wilf ome to light regarding the utility of using models in the development and ae 


evaluation of instructional products. 
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The Status of Research on Models of Product Development and 
Evaluation. By Nick L. Smith and Stephen L. Murray, Northwest 


Regional Educational Laboratory. $1.50 per copy (806-5201) 


Techniques and Procedures for Formative Evaluation. By James 
Sanders, Northwest Regional Educational Laboratory and Donald 


J. ‘Cunningham, Indiana University.- $3.00 per copy (806-5202) 


A Look at the Mosaic of Educational Evaluation and “Accountability, 
By Blaine Worthen, Northwest Regional Educational Laboratory. 


$2.50 per copy (806-5203) 
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The Administrator-Researcher Interaction: The Conduct of 
. Cooperative Research. By Stephen L. Murray and Nick L. Smith, 


Northwest Regional Educational Laboratory. $1.50 per copy 
(806-5204) 


Pre-product Evaluation: What to do While Waiting for the 
Product. By Nick L. Smith, Northwest Regional Educational 
Laboratory. $1.50 per copy (806-5205) 


A Prescriptive Model of Developmént or Evaluation: Some 
Needed Maturity. By Cecil Clark, Brigham Young University.” 


$2.50 per copy (806-5206) 


Some Tasks and Gompatenclen Frequently Required in Educational 
Research and Evaluation. By Blaine Worthen, Northwest see 


Educational Laboratory. $2.50 per copy (806-5207) z 


The Use of Multiple Strategies in the Evaluation of Experience~ 
Based Career Education?’ By Thomas R. Owens, Joseph F. Haenn and F 
Harry L. Fehrenbacher, Northwest Regional Educational Laboratory: 

$2.50 per copy (806-5208) 


The Use of Student Case Study Methodology ia Program Evaluation. 
By Harry L. Fehrenbacher, Thomas R. Owens and Joseph F. Haenn, 


Northwest Regional Educational Laboratory. $2.50 per copy 
(806-5209) 


Complete. Set.of Titles: $17.00 (reflects 15% discount) 


Prices subject to change without notice. 


Send purchase order or prepayment for, any of the titles listed above to: 


Office of Marketing ‘and Dissemination 
Northwest Regional Educational Laboratory 

710 S.W. Second Avefue 

Portland, Oregon 97204 
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